• DOMAIN: Automobile¶
• CONTEXT: The purpose is to classify a given silhouette as one of three types of vehicle, using a set of features extracted from the silhouette.¶
The vehicle may be viewed from one of many different angles.
• DATA DESCRIPTION: The data contains features extracted from the silhouette of vehicles in different angles. Four "Corgie" model vehicles were used for the experiment: a double decker bus, Cheverolet van, Saab 9000 and an Opel Manta 400 cars. This particular combination of vehicles was chosen with the expectation that the bus, van and either one of the cars would be readily distinguishable, but it would be more difficult to distinguish between the cars.¶
• All the features are numeric i.e. geometric features extracted from the silhouette.
• PROJECT OBJECTIVE: Apply dimensionality reduction technique – PCA and train a model using principal components instead of training the model using just the raw data.¶
In [2]:
import pandas as pd
import os
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
#from sklearn.preprocessing import StandardScaler,LabelEncoder
#from scipy import stats
%matplotlib inline
sns.set_style('darkgrid')
%matplotlib inline
#from sklearn.preprocessing import MinMaxScaler
from scipy.stats import zscore
from sklearn.model_selection import train_test_split
#from sklearn.neighbors import KNeighborsClassifier
from sklearn.metrics import accuracy_score, confusion_matrix
#from sklearn.linear_model import LogisticRegression
#from sklearn.tree import DecisionTreeClassifier
#from sklearn.naive_bayes import GaussianNB
from sklearn.model_selection import StratifiedKFold
from sklearn.svm import SVC
from sklearn.metrics import classification_report
from sklearn import model_selection
import warnings
warnings.filterwarnings("ignore")
#import plotly
#plotly.offline.init_notebook_mode()
#import plotly.graph_objs as go
#import plotly.tools as tls
#import plotly.figure_factory as ff
#from imblearn.over_sampling import RandomOverSampler
#from imblearn.over_sampling import SMOTENC
#from imblearn.over_sampling import SMOTE
from sklearn.model_selection import GridSearchCV
#from sklearn.ensemble import BaggingClassifier
#from sklearn.ensemble import RandomForestClassifier
#from sklearn.ensemble import GradientBoostingClassifier
#from sklearn.ensemble import AdaBoostClassifier
#from statistics import mean
#from tkinter import *
#import tkinter as tk
#from tkinter import ttk
#import pickle
#from sklearn.neighbors import KNeighborsClassifier
from sklearn.decomposition import PCA
In [3]:
os.chdir('C:\\Users\\VGopalak\\Desktop\\ML\\un-supervised learning')
Import, Clean and Pre-Process Data¶
In [62]:
df=pd.read_csv('Part3 - vehicle.csv')
Data Exploration
In [63]:
df.head() #sample 5 records.
Out[63]:
We can see all details related to vehicle is given.
In [6]:
df.shape #exploring shape of the dataset
Out[6]:
We have 846 rows and 19 columns.
In [7]:
df.columns #exploring column names
Out[7]:
In [8]:
df.info() #getting information about data
Missing value check
In [11]:
df.isnull().sum()
Out[11]:
We have very few missing values
Data Cleansing¶
Missing Value Treatment
Filling Missing value with median
In [64]:
null_col=[]
for i,j in enumerate(df.columns):
if df.isnull().sum()[i]>0:
df[j].fillna(df[j].median(),inplace=True)
null_col.append(j)
else:
pass
#print(null_col)
In [13]:
df.dtypes
Out[13]:
we have int,float and object datatypes
In [15]:
df.describe().T
Out[15]:
Below variables has high maximum value compared to 75% of data
- skewness_about.1,skewness_about,scaled_radius_of_gyration.1,scaled_radius_of_gyration,scaled_variance.1,scaled_variance,max.length_aspect_ratio,pr.axis_aspect_ratio,radius_ratio
Scale of the variables are different. We need to do scaling of data.
Outliers Treatment
Filling ouliers with mean(mean without outliers)
In [65]:
for c in df.select_dtypes(include=['int64','float64']).columns:
#getting upper lower quartile values
q25,q75=np.percentile(df[c],25),np.percentile(df[c],75)
IQR=q75-q25
Threshold=IQR*1.5
lower,upper=q25-Threshold,q75+Threshold
Outliers=[i for i in df[c] if i < lower or i > upper]
#taking mean of a column without considering outliers
df_include = df.loc[(df[c] >= lower) & (df[c] <= upper)]
mean=int(df_include[c].mean())
#imputing outliers with mean
df[c]=np.where(df[c]>upper,mean,df[c])
df[c]=np.where(df[c]<lower,mean,df[c])
Outliers=[i for i in df[c] if i < lower or i > upper]
EDA¶
Univariate Analysis¶
Distribution of numerical variables
In [17]:
outlier_col=[]
for i in df.select_dtypes(include=['int64','float64']).columns:
f, axes = plt.subplots(1, 2, figsize=(17,7))
sns.boxplot(x = i, data=df, orient='h' , ax=axes[1])
#sns.set(axis_bgcolor='k')
sns.distplot(df[i], ax=axes[0])
axes[0].set_title('Distribution plot of {}'.format(i))
axes[1].set_title('Box plot of {}'.format(i))
plt.show()
#checking count of outliers.
q25,q75=np.percentile(df[i],25),np.percentile(df[i],75)
IQR=q75-q25
Threshold=IQR*1.5
lower,upper=q25-Threshold,q75+Threshold
Outliers=[i for i in df[i] if i < lower or i > upper]
if len(Outliers)>0:
outlier_col.append(i)
print('{} Total Number of outliers in {}: {}'.format('\033[1m',i,len(Outliers)))
- Most of the variables have normal distribution.
- We can see two kinds of distribution in most of the variables.
- Since we hvae handled the outliers there is no outliers exists in data.
Correlation between numerical Variables.
In [18]:
plt.figure(figsize=(15,8))
sns.heatmap(df.corr(), annot=True, linewidths=.5, fmt= '.1f', center = 1 ) # heatmap
plt.show()
- We can see many variables are coorelated with each other.
- We can see few set of variables are even have value as 1
Distribution of Target Variable
In [19]:
df['class'].value_counts()
Out[19]:
In [20]:
f,axes=plt.subplots(1,2,figsize=(17,7))
df['class'].value_counts().plot.pie(autopct='%1.1f%%',ax=axes[0])
sns.countplot('class',data=df,ax=axes[1],order=['car','bus','van'])
axes[0].set_title('Class Variable Pie Chart')
axes[1].set_title('Class Variable Bar Graph')
plt.show()
- Car distribution is more compared to others.It has 50% of data
- Bus and van has nearly equal distribution.
Independent variables vs Dependent variables.
Checking mean values of each column with respect to target.
In [21]:
for i in df.select_dtypes(include=['int64','float64']).columns:
df.groupby(by=['class'])[i].mean().reset_index().sort_values([i]).tail(10).plot(x='class',
y=i,
kind='bar',
figsize=(15,5))
plt.show()
- There is no huge difference in mean values of each vehicle type
- Few variables has mean value higher for car and in few cases for bus.
- It is difficult to interpret which variable is important to particular class.
In [64]:
sns.pairplot(df,diag_kind='kde',diag_kws={'bw': 0.2})
Out[64]:
- We can see most of variables have positive co-relation with each other.
- Some Features are not correlated at all.
- We need to come up with efficient way of feature selection to build the model.
Classifier¶
Model Building with all features¶
In [22]:
df.dtypes
Out[22]:
In [23]:
df['class'].value_counts()
Out[23]:
- We will encode target variables before building the model.
In [66]:
df['class'].replace({'car':0,'bus':1,'van':2},inplace=True)
df['class']=df['class'].astype('category')
Splitting the dataset.
In [67]:
X=df.drop('class',axis=1)
y=df['class']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=1)
Grid Search CV.
In [68]:
# defining parameter range
param_grid = {'C': [0.1, 1, 10, 100, 1000],
'gamma': [1, 0.1, 0.01, 0.001, 0.0001],
'kernel':['rbf']}
grid = GridSearchCV(SVC(random_state=1), param_grid, refit = True)
# fitting the model for grid search
grid.fit(X_train, y_train)
print(grid.best_params_)
- We have found best parameters using Grid search.
In [70]:
X=df.drop('class',axis=1)
y=df['class']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.30, random_state=1)
svc = SVC(C=100,gamma=0.0001,kernel='rbf')
svc.fit(X_train, y_train)
predictions = svc.predict(X_test)
print(classification_report(y_test, predictions))
print("Accuracy on training set: {:.3f}".format(
svc.score(X_train, y_train)))
print("Accuracy on test set: {:.3f}".format(svc.score(X_test, y_test)))
#score_test=svc.score(X_test, y_test)
- Recall value for each class seems good.
We will train our model in multiple folds and find out the mean score
In [71]:
def score_model(model,params,cv):
#smote=SMOTE(random_state=1)
train_score=[]
test_score=[]
for train_fold_index,test_fold_index in cv.split(X,y):
X_train,X_test=X.iloc[train_fold_index],X.iloc[test_fold_index]
y_train,y_test=y.iloc[train_fold_index],y.iloc[test_fold_index]
#X_train_res,y_train_res=smote.fit_resample(X_train,y_train)
SVM=model(**params).fit(X_train,y_train)
y_pred=SVM.predict(X_test)
train_score.append(SVM.score(X_train,y_train))
test_score.append(SVM.score(X_test,y_test))
print('Train_score:',np.mean(np.array(train_score)))
print('Test_score:',np.mean(np.array(test_score)))
return np.mean(np.array(test_score))
In [72]:
cv=StratifiedKFold(n_splits=10,random_state=1)
params={'C': 100, 'gamma': 0.0001, 'kernel': 'rbf'}
b=score_model(SVC,params,cv)
We are getting 99% mean accuracy in training data and 98% mean accuracy in test data. This is a good model.
Dimensional Reduction.¶
PCA¶
We will use PCA to reduce the number of dimensions.
In [32]:
df.shape
Out[32]:
Initially we have 18 independent columns and one dependent column.
Covariance Matrix
In [34]:
covMatrix = np.cov(X,rowvar=False)
print(covMatrix.shape)
Covariance indicates the level to which two variables vary together.
Building PCA
Scaling the data
In [74]:
li=list(df.select_dtypes(include=['int64','float64']).columns)
print(li)
In [75]:
X[li]=X[li].apply(zscore) #replacing each continuous values with z-score value
In [76]:
pca = PCA(n_components=18) #initially we will create 18 components
pca.fit(X)
Out[76]:
Eigen Values
In [77]:
print(pca.explained_variance_)
We can see 18 eigen values which is corresponds to each principal component. Eigen values tells maximum ariance explained by each principal component.
Eigen Vectors
In [78]:
print((pca.components_.shape))
We have built 18 Principal components.
Explained Variance Ratio
In [79]:
print(pca.explained_variance_ratio_)
In [80]:
plt.bar(list(range(1,19)),pca.explained_variance_ratio_, align='center')
plt.ylabel('Variation explained')
plt.xlabel('eigen Value')
plt.show()
In [81]:
plt.plot(np.cumsum(pca.explained_variance_ratio_))
print(np.cumsum(pca.explained_variance_ratio_))
We can see 8 components explained 97% of variance in the data.
Dimensionality Reduction
Now 8 dimensions seems very reasonable. With 8 variables we can explain over 97% of the variation in the original data!
In [82]:
pca4 = PCA(n_components=8)
pca4.fit(X)
Xpca4 = pca4.transform(X)
In [50]:
sns.pairplot(pd.DataFrame(Xpca4))
Out[50]:
We can see all the features are not correlated with each others.This is because each principal component is perpendicular to each other.
Classifier¶
Fitting SVM With Reduced Attributes
In [83]:
X_train, X_test, y_train, y_test = train_test_split(Xpca4, y, test_size=0.30, random_state=1)
svc = SVC(C=1000,random_state=1)
svc.fit(X_train, y_train)
predictions = svc.predict(X_test)
print(classification_report(y_test, predictions))
print("Accuracy on training set: {:.3f}".format(
svc.score(X_train, y_train)))
print("Accuracy on test set: {:.3f}".format(svc.score(X_test, y_test)))
Training accuracy is 1 we will adjust hyper parameter to overcome this issue.
Grid Search CV
In [84]:
# defining parameter range
param_grid = {'C': [0.1, 1, 10, 100, 1000],
'gamma': [1, 0.1, 0.01, 0.001, 0.0001],
'kernel': ['rbf']}
grid = GridSearchCV(SVC(random_state=1), param_grid, refit = True)
# fitting the model for grid search
grid.fit(X_train, y_train)
Out[84]:
In [85]:
print(grid.best_params_)
In [86]:
def score_model_pca(model,params,cv):
#smote=SMOTE(random_state=1)
train_score=[]
test_score=[]
for train_fold_index,test_fold_index in cv.split(Xpca4,y):
X_train,X_test=Xpca4[train_fold_index],Xpca4[test_fold_index]
y_train,y_test=y.iloc[train_fold_index],y.iloc[test_fold_index]
#X_train_res,y_train_res=smote.fit_resample(X_train,y_train)
SVM=model(**params).fit(X_train,y_train)
y_pred=SVM.predict(X_test)
train_score.append(SVM.score(X_train,y_train))
test_score.append(SVM.score(X_test,y_test))
print('Train_score:',np.mean(np.array(train_score)))
print('Test_score:',np.mean(np.array(test_score)))
return np.mean(np.array(test_score))
In [87]:
cv=StratifiedKFold(n_splits=10,random_state=1)
params={'C': 10, 'gamma': 0.1, 'kernel': 'rbf'}
a=score_model_pca(SVC,params,cv)
We are getting balanced score in train and test data.
Conclusion¶
In [88]:
result = pd.DataFrame({'SVC' : ['All attributes', '8 Principle components'],
'Accuracy' : [b,a]})
result
Out[88]:
- We are getting less accuracy in test data after attributes are reduced.
- We are getting 98% accuracy using 18 features and 94 % acuracy in 8 features. only 4% of accuracy is reduced but we droped 10 features. This will reduce the complexity of the model. In production if the data size size is large it will take more time to build the model so that difficulties is reduced by dimensionality reduction
- Dimensionality reduction helped us to reduced the features from 18 to 8.